Search for: All records

Recent years have seen an explosion in methodological work on combining causal effects estimated from observational and experimental datasets. Observational data have the advantage of being inexpensive and increasingly available from sources such as electronic health records, insurance claims databases, and online learning platforms. These data are representative of target populations, but because treatment assignments are not randomized, they suffer from unmeasured confounding bias. By contrast, as a consequence of randomization, experimental data yield unbiased causal effects. Yet experiments are costly, often involve relatively few units, and may incorporate stringent inclusion criteria that make the studied populations somewhat artificial. A challenge for researchers is how to integrate these two types of data to leverage their respective virtues. Over roughly the past 5 years, many novel approaches have been proposed. As in this review, we restrict our focus to techniques for integrating individual‐level experimental and observational data, without assuming all confounding variables are studied in the observational data. We first “locate” the problem by detailing important considerations from the causal inference and transportability literature. We next discuss three important research traditions that predate modern methodological work: meta‐analysis, Empirical Bayes shrinkage, and historical borrowing. In organizing the growing literature on data‐combination methods, we use a categorization involving five distinct approaches: auxiliary methods, control‐arm augmentation, debiasing, test‐then‐merge, and weighting. Within each category, we summarize recently proposed methodologies, highlighting the strengths and weaknesses of each. We conclude with a discussion of how practitioners might choose between competing approaches when conducting applied work. 

Abstract The increasing availability of passively observed data has yielded a growing interest in “data fusion” methods, which involve merging data from observational and experimental sources to draw causal conclusions. Such methods often require a precarious tradeoff between the unknown bias in the observational dataset and the often-large variance in the experimental dataset. We propose an alternative approach, which avoids this tradeoff: rather than using observational data for inference, we use it to design a more efficient experiment. We consider the case of a stratified experiment with a binary outcome and suppose pilot estimates for the stratum potential outcome variances can be obtained from the observational study. We extend existing results to generate confidence sets for these variances, while accounting for the possibility of unmeasured confounding. Then, we pose the experimental design problem as a regret minimization problem subject to the constraints imposed by our confidence sets. We show that this problem can be converted into a concave maximization and solved using conventional methods. Finally, we demonstrate the practical utility of our methods using data from the Women’s Health Initiative.